Ultrasound Trainer with Internal Optical Tracking

ABSTRACT

A training system to teach use of an ultrasound probe, the training system having a chamber defining an orifice, a shaft insertable into the orifice of the chamber, a marker positioned on the shaft at a distal end, a camera positioned to view the marker when inserted inside the chamber, and a processor operatively connected to the camera for processing a position and an orientation of the shaft based on the marker. The system provides a method for visualizing movement of the shaft from inside the chamber.

CROSS-REFERENCE TO RELATED APPLICATION

This patent application claims the benefit of U.S. Provisional PatentApplication Ser. No. 62/798,860, entitled “Ultrasound Trainer withInternal Optical Tracking,” filed Jan. 30, 2019, which application isincorporated in its entirety here by this reference.

BACKGROUND

Currently available ultrasound simulation solutions that deliverendolumenal ultrasound simulation utilize excessively complex, largephysical footprint, and expensive inertial tracking, or alternativemotion sensing, technologies. The latter motion-sensing options are notcompatible with individual user utilization due to practical (e.g.,large form factor) and cost considerations, thereby—limiting ultrasoundtraining options. The proposed invention would deliver 6-DOF simulatedendolumenal ultrasound probe movement using a compact form factor for ascalable individual user training solution.

SUMMARY

The present invention is directed towards a training system to teach useof a medical device, such as an ultrasound probe. The training systemcomprises a chamber defining an orifice; a shaft insertable into theorifice of the chamber, the shaft having a proximal end and a distalend; a marker positioned on the shaft at the distal end; a camerapositioned to view the marker when inserted inside the chamber; and aprocessor operatively connected to the camera for processing a positionand an orientation of the shaft based on the marker.

In some embodiments, the system may further comprise a motion limiterconnected to the chamber at the orifice.

In some embodiments, the system may further comprise a light source toilluminate the chamber.

In some embodiments, the chamber mimics a body or a body part.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a schematic diagram of an embodiment of the present invention.

FIG. 2 is a schematic diagram of an embodiment of the present invention

FIG. 3 is a flow diagram of an embodiment of the present invention.

FIG. 4 is a perspective view of an embodiment of the present invention.

FIG. 5 is a perspective view of an embodiment of the present invention.

FIG. 6 is a perspective view of an embodiment of the present invention.

FIG. 7 is a cross-sectional view of a chamber taken at plane A shown inFIG. 6.

FIG. 8 is the cross-sectional view shown in FIG. 7 showing movement ofthe shaft (as indicated by the broken lined shafts).

FIG. 9 is the cross-sectional view shown in FIG. 7 showing insertion andwithdrawal movement of the shaft (as indicated by the broken linedshafts).

FIG. 10 is the cross-section view shown in FIG. 7 showing movement ofthe shaft in a chamber containing filler material.

FIG. 11 is a cross-sectional view of a chamber mimicking a human head.

FIG. 12 is a cross-section view of a chamber having an incision.

DETAILED DESCRIPTION OF THE INVENTION

The detailed description set forth below in connection with the appendeddrawings is intended as a description of presently-preferred embodimentsof the invention and is not intended to represent the only forms inwhich the present invention may be constructed or utilized. Thedescription sets forth the functions and the sequence of steps forconstructing and operating the invention in connection with theillustrated embodiments. It is to be understood, however, that the sameor equivalent functions and sequences may be accomplished by differentembodiments that are also intended to be encompassed within the spiritand scope of the invention.

The invention of the present application is a training system 100 forthe mastery of ultrasound procedures, including, but not limited toendolumenal (or endoluminal) ultrasound procedures, also known asendoscopic ultrasound (EUS) medical procedures, transvaginal sonography(TVS) or OB-GYN ultrasonography, rectal endoscopic sonography (RES),transesophageal echocardiogram (TEE), endobronchial, intestinal,intravascular ultrasound (IVUS) or similar diagnostic techniques wherean imaging probe is inserted into a bodily lumen of a human or animalsubject.

With reference to FIGS. 1-3, in the preferred embodiment, the trainingsystem 100 comprises: a chamber 1 defining an orifice 6, a shaft 2insertable into the chamber 1 through the orifice 6, a marker 5 on theshaft 2, a camera 4 configured to view inside the chamber 1 towards theorifice 6, and a processor 3 operatively connected to the camera 4 toprocess information. In some embodiments, a motion limiter 9 can beattached to the chamber 1 at the orifice 6. In some embodiments, a lightsource 8 can be configured to illuminate inside the chamber 1.

The marker 5 is an indicium endowed with a distinctive appearance and aknown geometric structure (shape) that can be detected by the camera 4.For example, the indicium may be a printed pattern or other markingapplied to shaft 2. The marker 5 can be affixed rigidly to one end ofshaft 2 by way of an adhesive, other bonding solution, or any other typeof fastener, or may be printed directly on the surface of shaft 2. Insome embodiments, the marker 5 may be formed directly into shaft 2either during the manufacturing process of the shaft 2 or as apost-production modification.

The shaft 2 itself may be constructed of a rigid material or a flexiblematerial, as long as the flexure of the shaft 2 does not causesignificant distortion of the appearance of the marker 5. The shaft 2has a handle 12 and a distal end 10 opposite the handle 12 to mimictypical imaging probes, such as ultrasound probes. The distal end 10 ofshaft 2 bears the marker 5. As such, the distal end 10 is insertableinto the chamber 1 through the orifice 6. The distal end 10 of shaft 2where the marker 5 is located may be referred to as the “tip” of shaft2. Shaft 2 emulates an imaging probe being inserted into a bodily lumen.The tip of shaft 2 emulates the location of the detector or transducerin a real endolumenal imaging probe, such as a transvaginal ultrasoundprobe, a transesophageal ultrasound probe, or an intravascularultrasound probe. In order to increase the realism of the endolumenalultrasound trainer, the preferred embodiment molds the shaft 2 toresemble a particular type of endolumenal ultrasound probe, and theoutside of the chamber 1 to resemble a section of the human body or thebody of an animal. One may also choose to affix an marker 5 to a realclinical probe, while ensuring that the appearance of the probe, whichmay be made of a reflective type of material, does not disturb the viewof the optical marker 5 by the camera 4, introducing glare and otherundesirable optical artifacts.

In some embodiments, shaft 2 may have a tip 10 that can be bent, throughflexing or articulation, independently from the main shaft 2 as shown inFIG. 4. Marker 5 may be applied to the tip 10. The shaft 2 may have oneor more controllers 11 on its exterior to mechanically alter theposition or orientation of the steerable tip (and thus marker 5 affixedon it). Common controllers 11 known to those skilled in the art mayinclude knobs, joysticks, dials, push buttons, capacitive buttons or anyother tactile or electronic control system. The motion of the controller11 may be transmitted directly to the tip 10 by an intermediatemechanical component or the tactile control may send a signal to aseparate actuator to steer the tip 10 in response to the operator'sinput. Alternatively, the controller 11 may be part of the userinterface of an external device, which in turn will send a signal to theshaft 2 and actuate the tip 10. The controller 11 may also be virtualelements of a graphical user interface (GUI) displayed on a screen thatmay be part of the shaft 2 itself or part of an external computingdevice.

In some embodiments, at least one marker 5 may be placed on a rigidsection of shaft 2 upstream of the tip 10 (i.e., towards the handle),and at least one other marker 5 may be placed on the tip 10. Analgorithm can analyze independent observations of each marker 5 andcompute the position and orientation of tip 10 and the rigid section ofshaft 2 separately.

In some embodiments, if the controller 11 can transmit its state to theprocessor 3, one can simulate the presence of a tip 10 with a singlerigid tubular shaft 2 with one or more markers 5, but without requiringthe additional complexity of a mechanically steerable tip. In thelatter, the processor 3 computes the location of the imagined steerableextension by combining information from optical tracking and the stateof the controllers 11 without requiring a physical mechanism toarticulate the tip 10 of the shaft 2.

The chamber 1 is hollow with at least one inner wall 7 defining aninterior space. For example, a single curved wall 7 can be used tocreate a cylindrical or spherical shape, multiple flat walls 7 can beused to create sidewalls, ceilings, and floors, or a combination ofcurved and flat walls can be used to create other three-dimensionalcavities. The inner wall 7 should be constructed with an opaque materialthat limits the transmission of external light into chamber 1.Preferably, the inside of chamber 1 is constructed of a material that ismatte in appearance or alternatively is coated with a substance thatreduces its optical reflectivity. Preferably, the distal end 10 of shaft2 that goes inside chamber 1 should have an appearance that is matte anddistinct in color and texture from the appearance of marker 5 so as tocreate a detectable contrast. Chamber 1 may mimic various body parts,such as the lower body (see, FIGS. 5-10 and 12) such as the vaginal areaand rectal area, the head and neck (see, FIG. 11), and the like, wherephysicians are likely to probe with an imaging probe. In someembodiments, the chamber 1 may be a full-sized manikin commonly used formedical training.

In some embodiments, a motion limiter 9 is formed or attached to thechamber 1 at the orifice 6 and spans its circumference adding thicknessto its outer edge. The motion limiter 9 is positioned in such way as toexert mechanical resistance against shaft 2, thus constraining itsfreedom of motion when the shaft 2 is inserted into chamber 1 andmanipulated by the user. Portions of shaft 2 in between the distal end10 and the handle 12 may be configured to mate with a motion limiter 9,which mechanically constrains shaft 2 to a desired range of motion. Themotion limiter 9 may have a flexible rubber trim whose thickness is suchto provide tight contact with shaft 2 and limit its insertion viafriction and its lateral motion by means of its stiffness againstdeformation. In this embodiment the contact between the motion limiter 9and the shaft 2 should be tight enough, so that the user cannot changeorientation of the shaft 2 without deforming the shape of motion limiter9. Therefore, in this embodiment there is a direct correlation betweenthe elasticity (the ability to deform) and coefficient of friction ofthe motion limiter 9 and the haptics of the shaft 2. Alternatively, themotion limiter 9 may be a cone or other revolved surface of a rigid orsemi-rigid material whose profile is calculated appropriately so as toconstrain the lateral motion of shaft 2 within a desired solid angle.

The camera 4 faces inside the opaque chamber 1 in such a way that itmaintains a clear view of the marker 5 for the entire range of motion ofshaft 2 when inserted into the chamber 1. In some embodiments, thecamera 4 may be inside the chamber 1. If a single camera 4 cannotobserve the marker 5 for the entire range of motion of the shaft 2, thesystem can employ multiple cameras 4 without violating the spirit of theinvention. In some embodiments, the shaft 2 can have multiple distinctmarkers 5 to ensure that at least one of the markers 5 is always visibleby at least one of the cameras 4 for the entire range of motion of theshaft 2. Therefore, each marker 5 may be distinct from another marker 5.In some embodiments, a marker 5 may be on the chamber wall 7, which canserve as a reference point to determine movement of the marker 5 on theshaft 2.

The camera 4 may be operatively connected to a processor that analyzesthe visual information captured by the camera 4. The connection may beestablished with a wired or wireless connection using either a standardprotocol, such as USB, Thunderbolt, Bluetooth or Wi-Fi, or a customprotocol as is well known to those skilled in the art. The processor maybe a microcontroller placed inside the chamber 1, it may be placedoutside the chamber 1 at a remote location, or it may be part of a morecomplex computing device.

We refer to the combination of the frame data from the camera 4 and thealgorithm that runs on the processor as “optical tracking”.

The optical tracking described in this invention allows for full 6degrees-of-freedom (6-DOF) tracking (rotation in 3 spatial dimensionsand position in 3 spatial dimensions) of the shaft 2. However, adesigner may employ different types of motion limiters 9 to furtherconstrain the motion to, for example, rotation only (3-DOF), rotationand planar motion (5-DOF), rotation and penetration (4-DOF), rollrotation and penetration (2-DOF).

The camera 4 may operate in the visible spectrum or the infraredspectrum, and may support multiple colors or be monochromatic. Oneskilled in the art understands that the appearance of the marker 5, theopacity of the chamber 1, and the internal coating of the chamber 1 mustbe chosen in a way that conforms to the chosen spectrum of light.Furthermore, for the purpose of this invention, optical tracking can beachieved adequately if the optical assembly of the camera 4 has a fixedfocus, manually adjustable focus, electronically adjustable focus, orauto-focus, or any other variants of varifocal lenses that can resolvethe pattern on the marker 5 with sufficient visual acuity. The imagesensor of camera 4 can either employ rolling shutter or global shutter.

In some embodiments, the camera 4 may be able to measure depth (e.g.,RGBD camera) directly by way of stereo vision, time-of-flight imaging,structured light, or other operating principle known to those skilled inthe art. Alternatively, the camera 4 may be a device specificallydesigned to track the three-dimensional position and orientation of anelongated object such as the commercially available LEAP Motioncontroller. This embodiment enables applications where the shaft 2 canbe an arbitrary elongated object and does not necessarily requiremodification by affixing a marker 5 to it.

In some embodiments, a light source 8 may also be directed towards theinside of the hollow chamber 1. For example, a light source 8 may bemounted on a mechanical assembly of the camera, or mounted on one of thewalls 7 of the chamber 1, or embedded into or attached behind the walls7 for backlighting. The light source 8 is designed to provide controlledillumination of the marker 5 for the entire range of motion of the shaft2. In some cases, the designer may employ more than a single lightsource 8 to ensure uniform illumination of the marker 5 and/orelimination of shadows for the entire range of motion of shaft 2.

In the preferred embodiment, the system may be combined with an externalcomputing device 3 that runs a software ultrasound simulator similar,but not limited, to The SonoSim® Ultrasound Training Solution. Anultrasound simulator comprises at least a case library that contains oneor more medical cases of interest, a user interface, a variable imagethat resembles the appearance of an ultrasound image or other clinicalimaging modality, and optionally a virtual representation of a patientalong with a visualization of the imaging device being inserted in abodily lumen. The system described in this invention is connected bymeans of a wire or wirelessly to a computing device 3 that runs theultrasound simulator. The computing device 3 may either receive rawframe data directly from the camera 4 and run the algorithm to computethe position and orientation of the tip 10 of the shaft 2, or it mayreceive position and orientation information of the tip 10 of the shaft2 already computed by the system through a processor 3 embedded in theapparatus itself.

The computing device 3 transmits the position and orientation of the tip10 of the shaft 2 to the ultrasound simulator and, in turn, theultrasound simulator updates the visualization to display an ultrasoundimage 20 that corresponds to the exact spatial configuration of shaft 2as if it were a real endolumenal probe inserted into a real patient asshown in FIG. 5.

Additionally, if shaft 2 is endowed with controllers 11, the operatormay alter the state of the ultrasound simulator by interacting withcontrollers 11. In the latter, the shaft 2 must have the ability totransmit the state of the controllers 11 to the computing device 3. Forexample, the operator turn a knob to steer the tip 10 of the probe andthe corresponding simulated ultrasound image 20 in the ultrasoundsimulator, or may press a button to switch the selected case from theavailable case library.

In use, the training system monitors movement of a marker on a distalend of a shaft with a camera, wherein the distal end of the shaft isinserted inside a chamber through an orifice of the chamber; anddetermines a position and orientation of the shaft based on the movementof the marker with a processor operatively connected to the camera. Insome embodiments, movement of the shaft is restricted with a motionlimiter. The position and orientation of the shaft can be calculatedusing an algorithm or a look up table. The movable shaft 2 can bepartially inserted into the orifice 6 of the chamber 1 where the opticalcamera 4 and light source 8 reside. Once inserted, movement of the partof the movable shaft 2 that is outside the chamber 1 (i.e., the handle12) results with the movement of the part of the movable shaft 2 that isinside the opaque chamber 1 and that has marker 5 (i.e., the tip 10).The movable shaft 2 may be guided (constrained) by the motion limiter 9.The light source 8 illuminates the chamber 1 to allow movement of themarker 5 to be captured by the camera 4. The images of the marker 5captured by camera 4 can be processed by a computer processor 3 tocalculate corresponding movement of the movable shaft 2.

When an operator moves the shaft 2 by manipulating it from the outsideof the chamber 1, the shaft 2 transmits the motion of the operator tothe optical marker 5 that is rigidly attached to the end of the shaft 2hidden inside the chamber 1.

Camera 4 observes the marker 5 as it moves, and transmits its frame datato the processor 3. The processor 3 employs an algorithm that correlatesthe observations of the marker 5 and its perspective distortions to theposition and orientation of shaft 2. Further processing by means ofmathematical transformations known to those skilled in the art allowsthe algorithm to determine the exact position and orientation of thedistal tip 10 of the shaft 2 that is hidden inside the chamber 1 inthree-dimensional space when in use.

EXAMPLES Pelvic Trainer

In one embodiment, the system emulates transvaginal sonography (TVS)and/or rectal endoscopic sonography (RES) as shown in FIGS. 5-10 withthe shaft 2 mimicking a probe. The endolumenal ultrasound trainercomprises an external mold that mimics the shape of a pelvis that hidesan internal hollow chamber 1; a shaft 2 that resembles atransvaginal/transrectal probe; an orifice 6 that allows insertion ofthe probe 2 into the internal hollow chamber 1; one or more opticalmarkers 5 affixed to the tip 10 of the probe 2; one or more opticalcameras 4; and one or more sources of light 8. The optical camera 4acquires observations of the optical marker 5 and sends them directly toan external computation device 3 that hosts an ultrasound simulatorcapable of displaying transvaginal/transrectal sonographic images 20.The operator manipulates the probe as they would do in a real OB-GYN orRES clinical session with a real patient. In response to the operator'smotions, the ultrasound simulator on computing device 3 displays anultrasound image 20 that realistically emulates the image the operatorwould see if they placed the transvaginal/transrectal probe inside apatient at the same position and orientation inside the body of thepatient.

Marker 5 may be a small disc with a pattern portraying two fullyoverlapping but non-concentric circles of contrasting colors. The offsetbetween the circles breaks the symmetry of the pattern and ensures thateach observation of the optical marker 5 determines the pose of theprobe unambiguously. Alternatively the marker 5 may be a rectangulartile with a pattern portraying a collection of squares of contrastingcolor arranged in a non-symmetric fashion.

The algorithm running on the processor first isolates the pixelscorresponding to optical marker 5 based on their distinct appearance.The algorithm then analyzes the pixels corresponding to marker 5 todetermine how the lens of camera 4 has applied a perspective distortionto the observed shape of the marker 5. Given a set of camera parametersknown in advance, the perspective distortion alters the size and shapein ways that can be predicted accurately.

In general if the marker 5 is designed appropriately, there is only onepossible position and orientation of the marker 5 in three-dimensionalspace that matches the view of marker 5 seen by the camera 4. Thealgorithm calculates this position and orientation using techniquesknown to those skilled in the art.

The shape of orifice 6 and the channel that guides probe 2 into chamber1 act as a motion limiter 9 that mechanically constrains probe 2. Theorifice 6 and channel may be replaceable to simulate different kinds ofmedical procedures. For example, TVS procedures may require lessconstrained probe rotations than RES procedures. An extended motionlimiter 9 can be inserted inside the chamber 1 to mimic the internalanatomy of the cavity (patient-specific or procedure-specific).Furthermore, one may employ one or a multitude of filler material 20 tofill the empty volume inside the chamber 1 to provide physicalresistance to the probe's 2 motion and emulate the haptic feedback ofinserting a transvaginal or transrectal probe inside the body cavity.The filler material 20 may have the consistency of a deformable solid ora viscous fluid. For example, the filler material 20 can be an opticallytransparent and deformable material. In some embodiments, the fillermaterial 20 can be a plurality of small, loose particles packed insidethe chamber 1.

TEE Trainer

In one embodiment, the system emulates transesophageal echocardiogram(TEE) sonography as shown in FIG. 11 with the shaft 2 mimicking a probe.The trainer comprises a chamber 1 that mimics the shape of a human headwith a cavity that corresponds to the inside of the mouth; shaft 2resembles a real transesophageal probe and is inserted into chamber 1through an orifice 6 at the mouth; a source of light 8 and a camera 4are located inside chamber wall 7. Probe 2 has control knobs 11 thatmimic the real steering controls of a TEE probe. Probe 2 has a flexibletip 10 that is actuated mechanically in response to control knobs 11 andone or more optical markers 5 are affixed on it. The position of markers5 informs the penetration depth (advancing and withdrawing) of probe 2inside the mouth. If more than one marker 5 is used, the relativeposition of the markers 5 informs the angle at which the operator hassteered the flexible tip 10 via the knob (rotate back/forward, flex toleft/right, anteflex/retroflex steering). Camera 4 observes marker 5 andthrough the processor transmits the computed position and orientation ofprobe 2 to computing device 3 running a simulator capable of visualizingultrasound of images of TEE sonography.

IVUS Trainer

In one embodiment, the system emulates intravascular ultrasound (IVUS)sonography as shown in FIG. 12 with the shaft 2 mimicking a probe. Thetrainer comprises a chamber 1 that mimics the shape of a human body or asection of it that is relevant to a particular type of intravascularprocedure. A small orifice 6 emulates a puncture into the body and aninternal channel 14 leads to a small hollow chamber 1 on the inside.Probe 2 mimics the shape of a real IVUS probe and has the marker 5attached to its tip 10. Camera 4 is located at one end of internalchamber 1 and observes the marker 5 affixed at the tip of the probe 2.Camera 4 observes the marker 5 and through the processor transmits thecomputed position and orientation of the probe to computing device 3running a simulator capable of visualizing ultrasound images 20 of IVUSsonography.

The foregoing description of the preferred embodiment of the inventionhas been presented for the purposes of illustration and description. Itis not intended to be exhaustive or to limit the invention to theprecise form disclosed. Many modifications and variations are possiblein light of the above teaching. It is intended that the scope of theinvention not be limited by this detailed description, but by the claimsand the equivalents to the claims appended hereto.

What is claimed is:
 1. An ultrasound trainer, comprising: a) a chamberdefining an orifice; b) a shaft insertable into the orifice of thechamber, the shaft having a proximal end, a distal end opposite theproximal end, a flexible tip at the distal end, and a control knob tobend the flexible tip; c) a marker positioned on the shaft at the distalend; d) a camera positioned to view inside the chamber towards theorifice; e) a processor operatively connected to the camera forprocessing a position and an orientation of the shaft based on themarker; f) a motion limiter connected to the chamber at the orifice; g)a filler material inside the chamber; h) a light source to illuminatethe chamber, wherein the chamber mimics a body.
 2. An ultrasoundtrainer, comprising: a) a chamber defining an orifice; b) a shaftinsertable into the orifice of the chamber, the shaft having a proximalend and a distal end opposite the proximal end; c) a marker positionedon the shaft at the distal end; d) a camera positioned to view insidethe chamber; and e) a processor operatively connected to the camera forprocessing a position and an orientation of the shaft based on themarker.
 3. The ultrasound trainer of claim 2, further comprising amotion limiter connected to the chamber at the orifice.
 4. Theultrasound trainer of claim 3, further comprising a filler materialinside the chamber.
 5. The ultrasound trainer of claim 4, furthercomprising a light source to illuminate the chamber.
 6. The ultrasoundtrainer of claim 5, wherein the shaft comprises a flexible tip at thedistal end.
 7. The ultrasound trainer of claim 6, wherein the shaftcomprises a controller to bend the flexible tip.
 8. The ultrasoundtrainer of claim 5, wherein the chamber mimics a body.
 9. The ultrasoundtrainer of claim 8, wherein the body is a vaginal area.
 10. Theultrasound trainer of claim 8, wherein the body is a rectal area, 11.The ultrasound trainer of claim 8, wherein the body is a head and neckarea.
 12. The ultrasound trainer of claim 2, further comprising a fillermaterial inside the chamber.
 13. The ultrasound trainer of claim 2,wherein the shaft comprises a flexible tip at the distal end.
 14. Theultrasound trainer of claim 13, wherein the shaft comprises a controllerto bend the flexible tip.
 15. The ultrasound trainer of claim 2, furthercomprising a light source to illuminate the chamber.
 16. The ultrasoundtrainer of claim 2, wherein the chamber mimics a body.
 17. A method fortraining use of an ultrasound, comprising: a) monitoring movement of amarker on a distal end of a shaft with a camera, wherein the distal endof the shaft is inserted inside a chamber through an orifice of thechamber; b) determining a position and orientation of the shaft based onthe movement of the marker with a processor operatively connected to thecamera.
 18. The method of claim 17, further comprising restrictingmovement of the shaft with a motion limiter.
 19. The method of claim 17,wherein the position and orientation of the shaft is determined by adistortion of the marker.
 20. The method of claim 17, wherein theposition and orientation of the shaft is determined using a look-uptable.